1. Field of the Invention
The present invention relates to image data processing, and more particularly, to a method for loop-filtering image data to reduce a quantization effect and an apparatus therefor.
2. Description of the Related Art
Generally, picture encoding standards such as MPEG of the International Standardization Organization (ISO) and H.263 recommended by the International Telecommunication Union (ITU) adopt block-based motion estimation and discrete cosine transform (DCT) blocks. Most picture encoding standards utilize the DCT in 8xc3x978 pixel block units in order to pack information with a small number of transform coefficients. This block-based DCT scheme is based on the local spatial correlation properties of an image.
However, when the block-based image data is restored, considerable image deterioration, for example, blocking artifacts near the block boundary, corner outliers at the cross points of blocks, and ringing noise near the image edges, occurs. This is because the image data go through the DCT in 8xc3x978 pixel block units prior to quantization. Such deterioration is serious when the image is highly compressed.
Blocking artifacts are caused by grid noise generated along the block boundary in a relatively homogeneous area. The grid noise shows traces of the block-based process at the edges between blocks when the compressed data is displayed on a screen after being restored. Thus, the edges between blocks are noticeable. The corner outlier is generated where the corners of the 8xc3x978 blocks meet. Also, the ringing noise is a typical Gibb""s phenomenon occurring by truncation when the coefficients of high-frequency components of the DCT are quantized so as to highly compress the image. Due to the ringing noise, one can notice overlapping of images with a predetermined interval.
In order to reduce the blocking artifacts, the corner outliers and the ringing noise occurring when block-based coding is performed, several methods have been suggested. A single-adaptive filtering (SAF) method has been introduced in order to reduce the quantization effect of JPEG-decompressed images by Y. L. Lee, H. C. Kim, and H. W. Park (xe2x80x9cBlocking Effect Reduction of JPEG images by Signal Adaptive Filteringxe2x80x9d in Press IEEE Trans. On Image Processing, 1997]. B. Ramanurthi and A. Gersho have suggested a method in which a two-dimensional (2-D) filter is used to reduce the blocking artifact while a one-dimensional (1-D) filter is used to reduce staircase noise [xe2x80x9cNonlinear Space Variant Postprocessing of Block Coded Imagesxe2x80x9d, IEEE Trans. on ASSP, Vol. 34, No. 5, pp 1258-1267, 1986]. Also, an iterative image-recovery algorithm using the theory of projections onto convex sets (POCS) has been proposed by Y. Ynag, N. Galatsanos and A. Katsaggelos [xe2x80x9cProjection-Based Spatially Adaptive Reconstruction of Block-Transform Compressed Images,xe2x80x9d IEEE Trans. on Image Processing, Vol. 4., No. 7, pp 896-908, July 1995]. Also, an article entitled xe2x80x9cA Deblocking Algorithm for JPEG compressed Images Using Overcomplete Wavelet Representationsxe2x80x9d has been disclosed by Z. Xiong, M. T. Orchard and Y. Q. Zhang [IEEE Trans. Circuits System Video Technology, Vol. 7, No. 2, pp 433-437, 1997].
However, the above methods require complicated computations.
To solve the above problems, it is an object of the present invention is to provide a method for loop-filtering image data and an apparatus therefor, capable of encoding at a low bit rate without need for complicated computations, by which quantization effects such as blocking artifacts, corner outliers and ringing noise caused by block-based decoding can be reduced.
According to one aspect of the object, there is provided a loop-filtering method for reducing quantization effect generated when an image data is encoded and decoded, the method comprising the steps of: extracting a flag indicating whether the image data requires loop-filtering using the distribution of inverse quantized coefficients (IQCs) of an inverse quantized image data and a motion vector indicating the difference between the previous frame and the current frame; and filtering the image data corresponding to the flag by a predetermined method if the extracted flag indicates a need for the loop-filtering.
Preferably, in the extracting step, the flag is extracted from a discrete cosine transform (DCT) domain of each block if an image frame to be loop-filtered is an intra frame, and from a residual signal and a reference frame if the image frame to be loop-filtered is an inter frame.
Preferably, the flag comprises a blocking flag indicating whether reduction of blocking artifacts near block boundaries is required, and a ringing flag indicating whether reduction of ringing noise near image edges is required.
Preferably, in the case where the image data is an intra frame, the blocking flag of the intra frame comprises a horizontal blocking flag (HBF) and a vertical blocking flag (VBF), wherein the horizontal and vertical blocking flag extraction of the intra frame comprises the steps of: calculating DCT coefficients of the inverse quantized 8xc3x978 block after the compressed image data is inverse quantized; assuming that a pixel A (DC coefficient) is located at the top, far left corner of an 8xc3x978 block consisting of 64 pixels, a pixel B is located to the right of the pixel A, and a pixel C is located below the pixel A, setting the HBF and the VBF to xe2x80x9c1xe2x80x9d indicating a need for the loop-filtering if only the coefficients in pixels A are non-zero; setting the VBF to xe2x80x9c1xe2x80x9d indicating a need for the loop-filtering if only the top row of the inverse quantized 8xc3x978 block is non-zero; and setting the HBF to xe2x80x9c1xe2x80x9d indicating a need for the loop-filtering if only the far left column of the inverse quantized 8xc3x978 block is non-zero.
Preferably, in the extraction of the ringing flag from the intra frame, assuming that a pixel A (DC coefficient) is located at the top, far left corner of the 8xc3x978 block consisting of 64 pixels, a pixel B is located to the right of the pixel A, and a pixel C is located below the pixel A, ringing flag RF is set to xe2x80x9c1xe2x80x9d indicating a need for loop-filtering when any pixel other than the pixels A, B and C of the inverse quantized 8xc3x978 block has a non-zero coefficient.
Preferably, the blocking flag of the current inter frame comprises a horizontal blocking flag and a vertical blocking flag, wherein assuming that a reference frame comprises a predetermined number of reference blocks, a block of the reference frame predicted by a motion vector (MVx0, MVy0) of the block Ac0 of the current inter frame is a motion block X0, the horizontal and vertical blocking flag extraction on a block Ac0 of the current inter frame comprises the steps of: investigating the degree of overlapping between the motion block X0 and the reference blocks; bit-wise AND-operating the HBF and the VBF of the reference blocks having a predetermined number of overlapped pixels; and setting the result of the bit-wise AND operation as horizontal and vertical blocking flags of the block Ac0 of the current video object plane (VOP).
Preferably, the horizontal and vertical blocking flag extraction on the block Ac0 of the current inter frame further comprises the step of setting the blocking flag of the reference macro block as the blocking flag of the current macroblock if a motion vector of the current macro block is a skipped macro block having zero motion vector.
Preferably, the horizontal and vertical blocking flag extraction on the block Ac0 of the current inter frame further comprises the step of padding the blocking flag of the reference block after replicating the blocking flag.
Preferably, the ringing flag extraction on a block Ac0 of the current inter frame in the case where the image frame is an inter frame, comprises the steps of: setting a ringing flag RF of the current block Ac0 to xe2x80x981xe2x80x99 if any AC component of inverse quantized coefficients (IQCs) of a residual signal of the 8xc3x978 inter blocks is non-zero, and to xe2x80x980xe2x80x99 if all AC components of the IQCs are zero; and setting the RF to xe2x80x981xe2x80x99 when the current block is in an 8xc3x978 prediction mode for transferring a motion vector on the macro block.
Preferably, in the case where the image data is an inter frame, the ringing flag of the inter frame comprises a first ringing flag (RF0) and a second ringing flag (RF1), wherein extraction of the first and second ringing flags RF0 and RF1 on a block Ac0 of the current inter frame comprise the steps of: setting the first ringing flag RF0 of the current block Ac0 to xe2x80x981xe2x80x99 if only the DC components of inverse quantized coefficients (IQCs) of a residual signal of the 8xc3x978 inter macro blocks are non-zero, and otherwise, setting the RF0 of the current block Ac0 to xe2x80x980xe2x80x99; and setting the second ringing flag RF1 of the current block Ac0 to xe2x80x981xe2x80x99 if any AC component of the IQCs of the residual signal of the 8xc3x978 inter frames is non-zero, and otherwise, setting the RF1 of the current block Ac0 to xe2x80x980xe2x80x99.
Preferably, assuming that a predetermined block constituting a frame to be loop-filtered is a block I and a block adjacent to the block I is a block J, the difference between the current block I and the quantized previous block I is not greater than a predetermined value, and the difference between the current block J and the quantized previous block J is not greater than a predetermined value, deblocking filtering is not performed in the filtering step.
Preferably, the deblocking filtering for reducing blocking artifacts comprises the steps of: performing a first filtering for changing a predetermined number of pixel values near a horizontal block boundary between the blocks I and J if the RFs of the blocks I and J indicate no need for reducing the ringing noise, and both HBFs of the blocks I and J indicate a need for reducing the blocking artifacts; and comparing the difference between two adjacent pixels around the block boundary with the H.263 quantization factor QP if both RFs of the blocks I and J indicate no need for reducing the ringing noise and at least one of the HBFs of the blocks I and J indicates no need for reducing the blocking artifacts, or if at least one of RFs of the blocks I and J indicate a need for reducing the ringing noise, and changing a predetermined number of pixel values, which is less than that for the first filtering, if the difference between two pixels is less than the QP.
Preferably, the deblocking filtering for reducing blocking artifacts comprises the steps of: performing a first filtering for changing a predetermined number of pixel values near a block boundary between the blocks I and J if the RF of the block I indicates no need for reducing the ringing noise, and both blocks I and J are intra-blocks; and comparing the difference between two adjacent pixels around the block boundary with the H.263 quantization factor QP if the RF of the block I indicates no need for reducing the ringing noise and at least one of the HBFs of the blocks I and J indicates no need for reducing the blocking artifacts, and performing a second filtering for changing a predetermined number of pixel values, which is less than that for the first filtering, if the difference between two pixels is less than the QP; performing the first filtering if the RF of the block I indicates no need for reducing the ringing noise, neither the block I nor the block J are intra-blocks, a DC component of inverse quantized coefficient (IQCs) of a residual signal of the block I or J exists, and both HBFs of the blocks I and J are xe2x80x981xe2x80x99; performing the second filtering if the RF of the block I indicates no need for reducing the ringing noise, neither the block I nor the block J are intra-blocks, a DC component of inverse quantized coefficient (IQCs) of a residual signal of the block I or J exists, and at least one HBF of the blocks I and J is zero; and performing the second filtering if the RF of the block I indicates a need for reducing the ringing noise.
Preferably, the deblocking filtering for reducing blocking artifacts comprises the steps of: performing a first filtering for changing a predetermined number of pixel values near a block boundary between the blocks I and J if the second ringing flag RF1 of the block I indicates no need for reducing the ringing noise, either the block I or the block J is an intra-block and both HBFs of the blocks I and J indicate a need for reducing the blocking artifacts; comparing the difference between two adjacent pixels around the block boundary with the H.263 quantization factor QP if the RF1 of the block I indicates no need for reducing the ringing noise, either the block I or the block J is an intra-block, and at least one of the HBFs of the blocks I and J indicates no need for reducing the blocking artifacts, and performing a second filtering for changing a predetermined number of pixel values, which is less than that for the first filtering, if the difference between two pixels is less than the QP; performing the first filtering if the RF1 of the block I indicates no need for reducing the ringing noise, neither the block I nor the block J are intra-blocks, at least one of RF0s of the blocks I and J is xe2x80x981xe2x80x99 and both HBFs of the blocks I and J are xe2x80x981xe2x80x99; performing the second filtering if the RF1 of the block I indicates no need for reducing the ringing noise, neither the block I nor the block J are intra-blocks, at least one of RF0s of the blocks I and J is xe2x80x981xe2x80x99, and at least one of the HBFs of the blocks I and J is not 1; and performing the second filtering if the RF1 of the block I is non-zero.
Preferably, assuming that six pixels on the horizontal block boundary between the blocks I and J are A, B, C, D, E and F, wherein the pixels C and D are the nearest pixels on the horizontal block boundary, the pixels A and F are the farthest pixels thereon, and pixels B and D are middle pixels, 7-tab low pass filtering with coefficients (1,1,1,2,1,1,1) is performed for the six pixels during the first filtering, and the second filtering is performed on the pixels C and D, wherein two pixels C and D are replaced as C=C+(Dxe2x88x92C)/4 and D=D+(Dxe2x88x92C)/4 if the absolute value of the difference between the pixels C and D is smaller than the QP of the H.263.
Preferably, the deringing filtering for reducing the ringing noise in the filtering step is performed if the ringing flag indicates a need for the deringing filtering, and is not performed if the ringing flag indicates no need for the deringing filtering, and the deringing filtering comprises the steps of: (a) detecting horizontal and vertical edges of the image data; and (b) performing a two-dimensional (2-D) signal adaptive filtering on an 8xc3x978 block from which the ringing noise is to be reduced. In the step (a) for the horizontal edge detection, assuming that a pixel[m,n] is the current pixel, a pixel[m][n+1] is located to the right of the pixel[m,n], a pixel[m][nxe2x88x921] is located to the left of the pixel[m,n], the difference between the pixel[m][n] and the pixel[m][n+1] is A1, the difference between the pixel[m][n] and the pixel[m][nxe2x88x921] is A2, and QP is the H.263 quantization factor, when the conditional expression ((A1 greater than QP) and (A2 greater than QP)) or (A1 greater than 2QP) or (A2 greater than 2QP) is satisfied, the current pixel is detected as an edge pixel, and the edge map, Edge[m][n], becomes 1. Also, in the step (a) for the vertical edge detection, assuming that the current pixel is pixel[m][n], a pixel[m+1][n] is located above the pixel[m][n], the lower pixel is pixel[mxe2x88x921][n], the difference between the pixel[m][n] and the pixel[m+1][n] is Axe2x80x21, the difference between the pixel[m][n] and the pixel[mxe2x88x921][n] is Axe2x80x22, and QP is the H.263 quantization factor, when the conditional expression ((Axe2x80x21 greater than QP) and (Axe2x80x22 greater than QP)) or (Axe2x80x21 greater than 2QP) or (Axe2x80x22 greater than 2QP) is satisfied, the current pixel is detected as an edge pixel, and the edge map, Edge[m][n], becomes 1. In the step (b) for the signal adaptive filtering, a 4-connected filter window is applied to the 8xc3x978 block to determine whether the center pixel of the filter window is an edge pixel, and the filtering is performed if the central pixel is an edge pixel, and otherwise, a weighted filtering is performed.
Preferably, the deringing filtering for reducing the ringing noise in the filtering step is not performed if the difference between the current block I and the quantized previous block I is not greater than a predetermined value, and the difference between the current block J and the quantized previous block J is not greater than a predetermined value.
Preferably, the deringing filtering for reducing the ringing noise in the filtering step is performed if the ringing flag of the block I indicates a need for deringing filtering, and otherwise, the deringing filtering is not performed.
Preferably, the deringing filtering for reducing the ringing noise is performed when the block I is an intra block, when the block I is not an intra block and has an 8xc3x978 prediction mode, and when the block I is not an intra block and has no 8xc3x978 prediction mode and the RF1 of the block 1 indicates no need for the deringing filtering, and the deringing filtering is not performed when the RF1 of the block I indicates no need for the deringing filtering.
Preferably, the deringing filtering comprises the steps of: (a) detecting horizontal and vertical edges of the image data; and (b) performing a two-dimensional (2-D) signal adaptive filtering on the 8xc3x978 block from which the ringing noise is to be removed. In the step (a) for the horizontal edge detection, assuming that a pixel[m][n] is the current pixel, a pixel[m][n+1] is located to the right of the pixel[m][n], a pixel[m][nxe2x88x921] is located to the left of the pixel[m][n], the difference between the pixel[m][n] and the pixel[m][n+1] is A1, the difference between the pixel[m][n] and the pixel[m][nxe2x88x921] is A2, and QP is the H.263 quantization factor, when the conditional expression ((A1 greater than QP) and (A2 greater than QP)) or (A1 greater than 2QP) or (A2 greater than 2QP) is satisfied, the current pixel is detected as an edge pixel, and the edge map, Edge[m][n], becomes 1. Also, in the step (a) for the vertical edge detection, assuming that the current pixel is pixel[m][n], a pixel[m+1][n] is located above the pixel[m][n], a pixel[mxe2x88x921][n] is located below the pixel[m][n], the difference between the pixel[m][n] and the pixel[m+1][n] is Axe2x80x21, the difference between the pixel [m][n] and the pixel[mxe2x88x921][n] is Axe2x80x22, and QP is the H.263 quantization factor, when the conditional expression ((Axe2x80x21 greater than QP) and (Axe2x80x22 greater than QP)) or (Axe2x80x21 greater than 2QP) or (Axe2x80x22 greater than 2QP) is satisfied, the current pixel is detected as an edge pixel, and the edge map, Edge[m][n], becomes 1. In addition, in the step (b) for the signal adaptive filtering a 4-connected filter window is applied to the 8xc3x978 block to determine whether the center pixel of the filter window is an edge pixel, and the filtering is performed if the central pixel is an edge pixel, and otherwise, a weighted filtering is performed.
Preferably, for reducing corner outlier generated at the point where the corners of four blocks meet when the image data is block-based coded and decoded, the loop-filtering method further comprises the steps of: (a) detecting a corner outlier of an 8xc3x978 block of the inverse quantized image data; and (b) compensating for the detected corner outlier. In the step (a) for the detection, assuming that A0, B0, C0 and D0 are four pixels around the point where the four corners meet, and value[0], value[1], value[2] and value[3] are pixel values of the pixels A0, B0, C0 and D0 respectively, the average of the pixels values is set to (A+B+C+D+2)/4 0, the difference between each pixel value of the average is compared to the H.263 quantization factor QP, and the number of corner outlier candidate pixels is accumulated to detect the corner outlier if the difference is greater than QP. Also, in the step (b) for the compensation, assuming that A10 and A20 are pixels adjacent to the pixel A0 and A30 is a pixel diagonal to the pixel A0 in the same block of the pixel A0, and Axe2x80x20, A1xe2x80x20 and A2xe2x80x20 are the compensated values of the pixels Axe2x80x20, A1xe2x80x20 and A2xe2x80x20, respectively, when only one corner outlier candidate is detected and the detected candidate is the pixel A0 and the difference between the pixels A0 and A30 is less than 3QP/2, the corner outlier compensation is performed using the following equation:
Axe2x80x2=(4A+B+C+2D+4)/8
A1xe2x80x2=(Axe2x80x2+3A1+2)/4
A2xe2x80x2=(Axe2x80x2+3A2+2)/4,
and when the number of candidates is more than two, the candidate which has the largest difference from (A3+B3+C3+D3+2)/4 0 is selected, and corner outlier compensation is performed on that point in the same manner as in the case of only one candidate.
According to another aspect of the object, there is provided an apparatus for loop-filtering image data, capable of reducing quantization effect generated when an image data is encoded and decoded, the apparatus comprising: a flag extractor for extracting a flag indicating whether the image data requires loop-filtering, using distribution of inverse quantized coefficients (IQCs) of the image data, and a motion vector indicating the difference between the previous frame and the current frame; a deblocking filter for deblocking filtering the image data by investigating the blocking flag extracted by the flag extractor; a corner outlier compensator for detecting the corner outlier of the deblocking-filtered data and compensating for the detected corner outlier; and a deringing filter for deringing-filtering the corner outlier compensated data by investigating the ringing flag extracted by the flag extractor, and wherein the flags comprise a blocking flag indicating whether a need for reducing blocking artifacts near block boundaries exists, and a ringing flag indicating whether there is a need for reducing ringing noise near image edges.
The invention may be embodied in a general purpose digital computer that is running a program from a computer usable medium, including but not limited to storage media such as magnetic storage media (e.g., ROM""s, floppy disks, hard disks, etc.), optically readable media (e.g., CD-ROMs, DVDs, etc.) and carrier waves (e.g., transmissions over the Internet). Hence, the present invention may be embodied as a computer usable medium.
According to still another aspect of the present invention, there is provided a computer readable medium having embodied thereon a computer program for loop-filtering image data, capable of reducing quantization effect generated when the image data is block-based encoded and decoded, wherein the loop-filtering comprises the steps of: (a) extracting a flag indicating whether the image data requires loop-filtering using the distribution of inverse quantized coefficients (IQCs) of an inverse quantized image data and a motion vector indicating the difference between the previous frame and the current frame; and (b) filtering the image data corresponding to the flag by a predetermined method if the extracted flag indicates a need for the loop-filtering, wherein in the step (a) the flag is extracted from a discrete cosine transform (DCT) domain of each block if an image frame to be loop-filtered is an intra frame, and from a residual signal and a reference frame if the image frame to be loop-filtered is an inter frame.